Ruler: split oversized remote distributor writes - #16160
Merged
Merged
Conversation
Contributor
|
💻 Deploy preview deleted (Mimir). |
Contributor
|
🤖 Automated comment |
aknuds1
force-pushed
the
arve/ruler-split-remote-distributor-writes
branch
from
August 4, 2026 14:59
4434c5d to
b84ee28
Compare
aknuds1
force-pushed
the
arve/ruler-split-remote-distributor-writes
branch
from
August 4, 2026 16:16
b84ee28 to
2f0ab5f
Compare
4 tasks
aknuds1
force-pushed
the
arve/ruler-split-remote-distributor-writes
branch
5 times, most recently
from
August 5, 2026 07:08
08c87f4 to
bdec8a2
Compare
aknuds1
marked this pull request as ready for review
August 5, 2026 07:12
aknuds1
force-pushed
the
arve/ruler-split-remote-distributor-writes
branch
from
August 5, 2026 07:32
bdec8a2 to
b7df148
Compare
Remote ruler writes were previously sent as a single gRPC message, so large rule evaluation results could exceed the configured transport limit. Split Remote Write 1.0 requests by series at the gRPC maximum send size and send each part sequentially with independent retries. Preserve request ownership and ordering, expose the number of generated requests, document partial-write semantics, and fix the RW1 splitter's protobuf size accounting and capacity estimates. Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
aknuds1
force-pushed
the
arve/ruler-split-remote-distributor-writes
branch
from
August 5, 2026 07:44
b7df148 to
e3e3ac8
Compare
juliusmh
requested changes
Aug 5, 2026
| The ruler splits remote writes along series boundaries so that they fit within `ruler.distributor.grpc_client_config.max_send_msg_size`. | ||
| When compression is enabled, the ruler reserves a conservative amount of space for compression framing, so it can split a write slightly below the configured transport limit. | ||
| It sends the resulting requests sequentially, with independent timeouts and retries; each request also consumes a separate gRPC client rate-limit token when rate limiting is enabled. | ||
| Requests accepted before a later request fails aren't rolled back. |
Contributor
There was a problem hiding this comment.
[nit] I don't think there's a way to roll-back a write?
Contributor
Author
There was a problem hiding this comment.
I don't think there's any claim it's possible either, right? It's just clarified they're not going to be undone (rolled back).
| // | ||
| // The returned requests may still retain references to fields in the original WriteRequest, i.e. they are tied to its lifecycle. | ||
| func SplitWriteRequestByMaxMarshalSize(req *WriteRequest, reqSize, maxSize int) []*WriteRequest { | ||
| if maxSize <= 0 { |
Contributor
There was a problem hiding this comment.
[nit] I think it's safe to assume maxSize is a non-zero positive integer.
| estimatedTimeseriesPerPartialReq := (len(req.TimeseriesRW2) / estimatedPartialReqs) + 1 // +1 is to round up | ||
|
|
||
| newPartialReq := func() (*WriteRequest, int) { | ||
| r := &WriteRequest{ |
Contributor
There was a problem hiding this comment.
I know u didn't touch this - but do you mind using the newPartialWriteRequest func here too? This would fix the missing SkipLabelCountValidation issue here too. Note: you'd have to manually initialize TimeseriesRW2 here.
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
…te-distributor-writes
Co-authored-by: Julius Hinze <julius.hinze@grafana.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
juliusmh
approved these changes
Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Remote ruler writes were previously sent as a single gRPC message, so large rule evaluation results could exceed
ruler.distributor.grpc_client_config.max_send_msg_size. This PR derives a compression-safe effective payload limit, splits oversized Remote Write 1.0 requests at series boundaries, and sends the resulting requests sequentially. Each generated request has its own timeout and retry lifecycle.Successful earlier requests aren't rolled back if a later request fails, so a failed rule evaluation can leave partially written results. The ruler emits one float or native-histogram sample per result series for each evaluation. Consequently, an individual result series exceeding the effective split limit cannot be losslessly subdivided. If this occurs, increase
ruler.distributor.grpc_client_config.max_send_msg_sizeand configure the distributor'sserver.grpc_server_max_recv_msg_sizeto be at least as large. When compression is enabled, a series above the conservative effective limit may still succeed if its compressed payload fits within the configured transport limit; otherwise gRPC returnsResourceExhausted.The implementation preserves request ownership and series ordering and reuses the RW1 write-request splitter already used by ingest storage to keep Kafka records within their configured size limit. The ruler adds compression-aware sizing, sequential gRPC sends, and per-request retries around that splitter. The
cortex_ruler_remote_distributor_requests_per_write_requesthistogram reports how many remote requests each ruler write generates. This PR also fixes the RW1 splitter's protobuf embedded-message framing accounting and caps its initial preallocation for very small limits. The corresponding RW2 fixes are in #16295.Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]. If changelog entry is not needed, please add thechangelog-not-neededlabel to the PR.about-versioning.mdupdated with experimental features. Not applicable: this PR doesn't add an experimental feature or configuration surface.